User interface for image-based rendering of virtual tours

ABSTRACT

Under an embodiment of the invention, a computer system includes a user interface (UI) for virtual tours of modeled real estate. The UI includes three ways to navigate the modeled real estate. The first navigation element is a 3D UI element that provides a 3D view of the property and enables the user to freely navigate in the XY plane of the virtual model. The second navigation element is a 2D map of the real property overlaid on a portion of the 3D UI element. The 2D map overlay enables navigation in the XY plane of the virtual model. The third navigation element is a 1D text element of the real property overlaid on a portion of the 3D UI element. The 1D text overlay enables navigation along the Z axis or the XY plane of the virtual model.

This application claims the benefit of U.S. Nonprovisional patentapplication Ser. No. 14/525,052, filed Oct. 27, 2014, now Issues as U.S.Pat. No. 10,242,400 on Mar. 26, 2019, which claims the benefit of U.S.Provisional Patent Application No. 61/895,978, filed Oct. 25, 2013. Thisapplication is related to U.S. patent application Ser. No. 14/525,057,filed Oct. 27, 2014, entitled “IMAGE-BASED RENDERING OF REAL SPACES”,U.S. patent application Ser. No. 14/525,059, filed Oct. 27, 2014,entitled “IMAGE-BASED RENDERING OF VIRTUAL MODELS OF REAL SPACES”, andU.S. patent application Ser. No. 14/525,060, filed Oct. 27, 2014,entitled “IMAGE-BASED RENDERING OF THREE DIMENSION GEOMETRIES”, filedconcurrently herewith and commonly assigned. The above U.S. patentapplications are fully incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates generally to methods and systems for image-basedrendering, such as user interfaces for image-based rendering of virtualtours of real spaces.

BACKGROUND

The typical user interface for virtual real estate tours ischaracterized by numerous inefficiencies that create navigation problemsfor users.

Potential home buyers suffer from a lack of real estate information andtools. The websites of many real estate brokerages generally providesome photographs of the properties in their listings. However, thesewebsites remain surprisingly poor at providing comprehensive visualinformation about properties.

From the buyer's perspective, real estate websites suffer from numerousproblems at present. First among these is the typical gallery ofphotographs of the property. Usually, the photographs are taken by thereal estate agent or homeowner and are intended to highlight thepositive attributes of the property while minimizing any negativeattributes. Second, the photo galleries are usually navigated linearly,by proceeding from one two-dimensional photograph to the next. Lastly,the viewer must usually guess at which location within the property thephoto is taken (for example, a photo of a master bedroom might leave theviewer guessing as to whether the master bedroom is in the front or backof the house, on the first floor or second floor, etc.).

There is a need for a system that overcomes limitations of the currentuser interface of real estate web sites, as well as providing additionalbenefits.

SUMMARY

A brief summary of some embodiments and aspects of the invention arefirst presented. Some simplifications and omissions may be made in thefollowing summary; the summary is intended to highlight and introducesome aspects of the disclosed embodiments, but not to limit the scope ofthe invention. Thereafter, a detailed description of illustratedembodiments is presented, which will permit one skilled in the relevantart to make and use aspects of the invention. One skilled in therelevant art can obtain a full appreciation of aspects of the inventionfrom the subsequent detailed description, read together with theFigures, and from the claims (which follow the detailed description).

Under an embodiment of the invention, a web site system maintains a realestate web page. The real estate web page facilitates three-dimensional(3D) image-based rendering virtual tours of real properties through aunique user interface that provides multiple viewpoints and tournavigation tools. The web site system facilitates virtual tours of realestate, such as homes, that are offered for sale via the web sitesystem. The web site system can store various information aboutproperties in which the user is interested and facilitate sharing ofinformation with a real estate agent or other service provider (e.g., alocal merchant).

One embodiment of the disclosed invention includes a method, embodied ina computer system, of providing virtual tours of real property. Themethod includes the steps of maintaining a web site system thatfacilitates selection of a real property from a database of realproperties; receiving from a user a first selection, the first selectionindicating a real property which the user would like to virtually tour;receiving from the user a request to initiate a virtual real estatetour; displaying a 3D viewpoint within a virtual model of the realproperty; displaying a 2D map overlay of the real property on a firstportion of the user interface; and displaying a 1D text overlay of thereal property on a second portion of the user interface.

One embodiment of the disclosed invention is a system for providingimage-based rendering of real property. The system including a servercomputer for maintaining a website system, the server operativelyconnected to a database of real properties; the website systemfacilitating selection of a real property from the database of realproperties; and a user interface, operatively connected to the websitesystem and the server computer. The system receiving a first selectionfrom a user, the first selection indicating a real property which theuser would like to virtually tour; receiving a second selection from theuser, the second selection initiating a virtual real estate tour;displaying a 3D viewpoint within a virtual model of the real property;displaying a 2D map overlay of the real property on a portion of theuser interface; and displaying a 1D text overlay of the real property ona portion of the user interface.

One embodiment of the disclosed invention includes a non-transitorycomputer readable medium containing a program adapted to implement amethod including the steps of presenting a webpage to a user; receiving,via the webpage, a first selection from the user, the first selectionindicating a real property which the user would like to virtually tour;receiving from the user a request to initiate a virtual real estatetour; displaying a 3D UI element within a virtual model of the realproperty, the 3D UI element enabling navigation in the XY plane of thevirtual model; displaying a 2D map overlay of the real property on aportion of the 3D UI element, the 2D map overlay enabling navigation inthe XY plane of the virtual model; and displaying a 1D text overlay ofthe real property on a portion of the 3D UI element, the 1D text overlayenabling navigation along the Z axis or the XY plane of the virtualmodel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a personal property merchandising andadvertising system suitable for use with the disclosed inventions.

FIG. 2 shows a block diagram of an embodiment of a computer systemsuitable for use with the disclosed inventions.

FIG. 3 shows a virtual tour user interface according to one embodimentof the invention.

FIG. 4 shows a block diagram of a virtual tour user interface accordingto one embodiment of the invention.

FIG. 5 shows a block diagram of a virtual tour user interface accordingto one embodiment of the invention.

FIGS. 6A and 6B show a block diagram of a virtual tour user interfaceaccording to one embodiment of the invention.

FIGS. 7A and 7B show a block diagram of a virtual tour user interfaceaccording to one embodiment of the invention.

FIG. 8 shows a block diagram of a virtual tour user interface accordingto one embodiment of the invention.

FIG. 9 shows a block diagram of a virtual tour user interface accordingto one embodiment of the invention.

FIG. 10 shows a block diagram of a transaction flowchart suitable foruse with the disclosed inventions.

FIG. 11 shows a virtual tour user interface according to one embodimentof the invention.

FIG. 12 shows a flowchart of an embodiment of a method of providinginformation about virtual tour property suitable for use with thedisclosed inventions.

The headings provided herein are for convenience only and do notnecessarily affect the scope or meaning of the claimed invention.

In the drawings, the same reference numbers and acronyms identifyelements or acts with the same or similar functionality for ease ofunderstanding and convenience. To easily identify the discussion of anyparticular element or act, the most significant digit or digits in areference number generally refers to the Figure number in which thatelement is first introduced (e.g., element 102 is first introduced anddiscussed with respect to FIG. 1).

DETAILED DESCRIPTION

The following description provides specific details for a thoroughunderstanding of, and enabling description for, these embodiments of theinvention. However, a person of ordinary skill in the art willunderstand that the invention may be practiced with many variations andthese details do not list every possible variation. In some instances,well known structures and functions have not been shown or described indetail to avoid unnecessarily obscuring the description of theembodiments of the invention.

Overview

In some embodiments of the invention, a web site implemented in acomputer system allows users to navigate a virtual tour of a realproperty. Among its many uses described herein, embodiments of the website system provide a capability to tour a real property withoutphysically visiting the property, to store and share portions of virtualtours, and to search for objects to purchase from merchants. In someembodiments, the system displays information on the web page but otherdelivery methods can be used, such as streaming video or email.

Embodiments of the invention include numerous innovative informational,analytical, and collaborative tools. Geometry and image data arecombined to create photorealistic synthetic views using different IBRapproaches, based on the density of the data and geometry. The systemmay deliver the data by client-server over the Internet. Photorealismmay be maintained within panoramas at image capture locations, which arethe primitives in some embodiments of the system.

Some embodiments of the user interface provide valuable data that isaggregated from system user trends. Thus property sellers can see howmany system users have viewed their virtual property, what areas of thevirtual tour were most interesting to the viewers, and so on. On anindividual level, this “backend” data can also be used for targetedadvertising. For example, if a visitor to the virtual property isviewing the kitchen, the web site system might show an advertisement foran appliance store from which the refrigerator in the virtual kitchenmay be purchased. Similarly, the system might show an advertisement fora furniture store that specializes in furniture in a style similar tothat in which the virtual property is decorated.

In some embodiments, the system includes mechanisms for collaborationbetween various users of the system. Shared access to select useraccount information and a messaging system allow users to share their“favorites” folder with their real estate agents, leave comments andquestions for home sellers, receive “suggested” property tours in theusers “suggested” folder, and append comments or notes that will bevisible to the user. In various embodiments, the messaging system maycontain user accounts having instant messaging, email, and folders forstoring and sharing favorites using any of well-known techniques forimplementing those features.

Example Computing Environment for Providing User Interfaces forImage-Based Rendering

FIG. 1 shows a block diagram of an exemplary embodiment of a computingenvironment providing the systems and methods described herein. A usermay access a virtual tour user interface over the Internet 106 via awebpage or streamed media hosted on a remote server 110. Alternatively,the UI may be hosted locally on smartphone 102 or mobile computer 104,either as a standalone application or cached media. When the userselects an object shown in virtual tour, a search engine, which may behosted on the server computer 110, will query the 3D object imagedatabase 112 to identify the object by matching its 3D image. Image andtext recognition software will extract any trademark or text (e.g.,manufacturer name) from the object's 3D image. The search engine willquery the trademark image database 114 and/or the trademark textdatabase 116, depending on whether trademark text or image weresuccessfully extracted from the object image. After the object has beenidentified, the server computer 110 will query advertising database 18and merchant database 120 to identify relevant advertisements andmerchants that sell the object. In the embodiment shown, the servercomputer 110 will then provide the product, merchant, and advertisinginformation to the user via the user interface. Although the product andmerchant identification process has primarily been discussed in thecontext of an on-demand system without metadata or product tagsassociated with objects in the virtual tour, it is important to notethat alternative embodiments may use such metadata to identify productsrather than relying on image matching and trademark searching. Forexample, the product tag associated with the chair in UI element 1110 ofFIG. 11 could indicate that product and merchant information about thechair have been preloaded into the virtual tour data.

FIG. 2 and the following discussion are intended to provide a brief,general description of a suitable computing environment 200 in whichvarious aspects of the disclosed invention may be implemented. Thoseskilled in the art will recognize that the invention also may beimplemented as a combination of hardware and software. Generally,program modules include routines, programs, components, data structures,etc., that perform particular tasks or implement particular abstractdata types. Moreover, those skilled in the art will appreciate that theinventive methods may be practiced with other computer systemconfigurations, including single-processor or multiprocessor computersystems, minicomputers, mainframe computers, as well as personalcomputers, hand-held computing devices, microprocessor-based orprogrammable consumer electronics, and the like, each of which may beoperatively coupled to one or more associated devices. Aspects of thedisclosed inventions may also be practiced in distributed computingenvironments where certain tasks are performed by remote processingdevices that are linked through a communications network. In adistributed computing environment, program modules may be located inboth local and remote memory storage devices.

FIG. 2 shows an exemplary environment 200 for implementing variousaspects of the disclosed inventions that includes a computer 202, thecomputer 202 including a processing unit 204, a system memory 206 and asystem bus 208. The system bus 208 couples system components including,but not limited to, the system memory 206 to the processing unit 204.The processing unit 204 may be any of various commercially availableprocessors. Dual microprocessors and other multi-processor architecturesmay also be employed as the processing unit 204.

The system bus 208 can be any of several types of bus structure that mayfurther interconnect to a memory bus (with or without a memorycontroller), a peripheral bus, and a local bus using any of a variety ofcommercially available bus architectures. The system memory 206 includesread only memory (ROM) 210 and random access memory (RAM) 212. A basicinput/output system (BIOS) is stored in a non-volatile memory 210 suchas ROM, EPROM, EEPROM. A BIOS contains the basic routines that help totransfer information between elements within the computer 202, such asduring start-up.

The computer 202 further includes a hard disk drive 214. The hard diskdrive 214 can be connected to the system bus 208 by a hard disk driveinterface 216. The removable storage drives (DVD drives, floppy drives,etc.) are not shown for clarity. However, the removable storage drivesand their associated computer-readable media provide nonvolatile storageof data, data structures, and computer-executable instructions forimplementing the inventions described herein. For the computer 202, thedrives and media accommodate the storage of information input by a user,or received from a remote computer, in a suitable digital format.Although the description of computer-readable media above refers to ahard disk, a removable magnetic disk, and a DVD, a person of ordinaryskill in the art understands that other types of storage media which arereadable by a computer, such as zip drives, magnetic cassettes, flashmemory cards, digital video disks, cartridges, and the like, may also beused in the exemplary operating environment, and further that any suchmedia may contain computer-executable instructions for performing themethods of the present invention.

Software applications can be stored in the drives and RAM 212. Theseapplications can include an operating system 230, one or moreapplication programs 232, (e.g., web browsers and client applications,etc.) other program modules 234 (e.g., cookies, etc.) and program data236. All or portions of the operating system, applications, modules,and/or data can also be cached in the RAM 212.

Embodiments of the invention can be implemented with variouscommercially available operating systems or combinations of operatingsystems.

A user can enter commands and information into the computer 202 througha keyboard 244 and a pointing device, such as a mouse 242. For example,the user might employ the mouse to navigate the virtual tour userinterface. Other input devices (not shown) may include a microphone, anIR remote control, a joystick, a game pad, similar devices. These andother input devices are often connected to the processing unit 1204through a serial port interface 240 that is coupled to the system bus208, but may be connected by other interfaces, such as a parallel port,a game port, a universal serial bus (“USB”), an IR interface, a wirelesstransceiver 258, etc. A monitor 220 or other type of display device isalso connected to the system bus 208 via an interface, such as a videoadapter 218. In addition to the display 220, a computer typicallyincludes other peripheral output devices (not shown), such as speakers,printers, etc., that can present information to the user.

As shown in FIG. 1, the computer 202 may operate in a networkedenvironment using logical connections via wired and/or wirelesscommunications to one or more remote computers, such as a remotecomputer(s) 248. The remote computer(s) 248 may be a workstation, aserver computer, a router, a personal computer, portable computer,microprocessor-based entertainment appliance, a peer device or othercommon network node, and typically includes many or all of the elementsdescribed relative to the computer 202, although, for purposes ofbrevity, only a memory storage device 250 is illustrated. The logicalconnections depicted include a local area network (LAN) 252 and awireless local area network (WLAN) 254. Such networking environments arecommonplace in homes and businesses. The Internet can also be used toprovide access to remote computer 248.

When used in a LAN networking environment, the computer 202 is connectedto the local network 252 through a wired or wireless communicationnetwork interface or adapter 256. The adaptor 256 may facilitate wiredor wireless communication to the LAN 252. When used in a WLAN networkingenvironment, the computer 202 typically is connected to a communicationsserver on the LAN, or has other means for establishing communicationsover the WLAN 254, such as the Internet. In a networked environment,program modules depicted relative to the computer 202, or portionsthereof, may be stored in the remote memory storage device 250. Thenetwork connections shown are exemplary and other means of establishinga communications link between the computers may be used.

The computer 202 is operable to communicate with any other deviceshaving wireless communication capability, e.g., a cell phone, a printer,desktop and/or portable computer, portable data assistant, andtelephone. As discussed briefly above, suitable wireless technologiesmay include, but are not limited to, cellular, WLAN (e.g., IEEE 802.11),IEEE 802.16, IEEE 802.20, and Bluetooth.

IEEE 802.11, also commonly known as “Wifi”, is a wireless communicationprotocol that enables computers to send and receive data anywhere withinthe range of a base station. A WLAN can be used to connect computers toeach other, to the Internet, and to wired networks (which may use IEEE802.3 or Ethernet communication protocols).

Aspects of the invention described above may be stored or distributed oncomputer-readable media, including magnetic and optically readable andremovable computer discs, as well as distributed electronically over theInternet or over other networks (including wireless networks). Thoseskilled in the relevant art will recognize that portions or embodimentsof the invention may also reside in a fixed element of a communicationnetwork such as a server or database, while corresponding portions mayreside on a mobile communication device, such as a laptop computer,Personal Digital Assistant (“PDA”), or mobile phone. Data structures andtransmission of data particular to aspects of the invention are alsoencompassed within the scope of the invention.

In accordance with the practices of persons skilled in the art ofcomputer programming, embodiments of the invention are described withreference to acts and operations that are performed by computer systems.Such computer-executed acts and operations may be performed by anoperating system (e.g., Microsoft Windows, Linux, Apple iOS, Android) oran application program. The acts and operations include the manipulationby the CPU of electrical signals representing data bits and themaintenance of data bits at memory locations to operate the computersystems and process signals. The memory locations where data bits aremaintained are physical locations that have particular electrical,magnetic, or optical properties corresponding to the data bits. Althoughdatabases are shown as separate physical/logical entities for clarity,they may be combined and/or aggregated to suit the physical/logicalarchitecture of the system in which they are used.

Example User Interfaces for Image-based Rendering of Virtual Tours ofReal Spaces

FIG. 3 shows an embodiment of a user interface 300 for navigating athree-dimensional (3D) virtual tour of a real property. Three differentuser interface elements serve the dual purposes of informing the user ofhis location in the model and simultaneously enabling spatialnavigation. These three elements are shown in the user interface 300embodiment of FIG. 3, which would normally be contained within a browserwindow, within a framed client application, or as the entire screenduring full screen mode. User Interface element 302 is the viewpointwithin a virtual model generated by combining geometry and image datausing Image-Based Rendering (IBR), thus creating a 3-dimensional (3D)view. UI element 304 is a two-dimensional (2D) map overlay that displaysthe relative location in the virtual model of the current viewpoint 306shown in UI element 302. UI element 308 is a text overlay that displaysone or more labels associated with the user's location within thevirtual model.

FIG. 4 shows a block diagram of the three primary user interfaceelements of an embodiment of the navigation tool. UI element 302 shows arendering of the model from a specific position and also serves as ameans to navigate to adjacent positions and to change the viewing vectorfrom a fixed position. Navigation can occur using various well knowninput/output (10) devices, such as a keyboard, touchscreen, eye-trackingtechnology, gesture recognition technology, or a computer mouse. Fordensely sampled spherical panoramas, one example of navigation using UIelement 302 with a mouse would be to click on the rendered view totranslate in the XY plane to another panorama location in the model.Another example of navigation using UI element 302 would be to click andhold the mouse button, enabling rotation about the Z axes, thus “lookingaround” without translating in the XY plane. As the user navigates, therendered viewpoint shown in UI element 302 changes in real time based ona new position and viewing vector associated with the new location.

Note that UI elements 304 and 308 are shown in different positions onthe UI 300. In general, these UI elements 304 and 308 may berepositioned in any suitable configuration without affecting theunderlying functionality of the overall UI 300.

FIG. 5 shows a block diagram of the user interface 300 with a simplifiedhouse 502 rendered in UI element 302. In this Figure, UI element 302 isshowing a 3D view of the exterior of house 502. UI element 304 displaysa corresponding parcel map including house 502. The viewpoint indicator306 is also overlaid on the parcel map to indicate the viewpointdisplayed in UI element 302.

FIGS. 6A and 6B show a block diagram of an exemplary scenario involvingnavigation within a dense sampling of spherical panoramas. FIG. 6A showsan embodiment of UI 300. FIG. 6B shows an exemplary ground truth imagedata capture map corresponding to the virtual model in FIG. 6A. Lowdensity sampling is typically used outside of a house. High densitysampling is typically used inside the house and generally consists of atleast one sample per eighty square feet of floor space. UI element 302can be used to rotate with two degrees of freedom (2 DOF) inside of aspherical panorama or translate with 2 DOF in the XY plane to adjacentpanoramas. In FIG. 6A, user interface element 302 is shown with a 120degree view (v) from a panorama at position 5 in FIG. 6B. FIG. 4B showsa uniform distribution of captured panoramas at locations 1 through 9.In the embodiment shown of UI element 302 in FIG. 4A, there are 3adjacent panoramas to which the user can translate based on the field ofview (e.g., the field of view points towards panorama capture points 1,2, and 3, thus defining the translation possibilities for UI element 102absent a rotation that would encompass other capture points within thefield of view).

Clicking within UI element 302 on positions 1, 2, or 3 shown in FIG. 6A,transitions the viewpoint to spatial location of the similarly labeledpanorama capture points shown in FIG. 6B. The viewpoint transition mayuse a number of methods including a fade, directional blending, imageflow based warping, or transitions using full 3D information. Other userinterfaces are also possible including mapping of 2D screen interactionswith the actual 3D model for non-contiguous jumps involving translationand rotation based on the geometry of the model.

FIGS. 7A and 7B show a block diagram of an exemplary scenario whereinthe user interface will not permit the user to translate in thedirection of the current viewpoint v. For UI element 302 a connectivitygraph may determine if translation between dense spherical panoramas ispossible or not. A connectivity graph shows all of the data capturelocations on a 2D map of the property. If a transition is possiblebetween two capture points, the connectivity graph will indicate that atransition is available. When the connectivity graph does not enable atranslation from a particular viewpoint, UI element 302 may providefeedback that no translation movement is possible. In FIG. 7B, we see auser viewing vector from position 2 on the right. The user is looking ata wall in this position, and no translation movement is possible withoutfirst rotating. In FIG. 7A, UI element 302 shows a ‘stop sign’ 702,indicating that no translation is possible from this viewing vectorwithout rotating first.

FIG. 8 shows a block diagram of User Interface 300 with an enhanced viewof UI element 304. User Interface element 304 is a map overlay thatidentifies an XY position within the virtual model using simplifiedgeometry (such as a floor plan) to show the current viewpoint's relativelocation in the model—this is a 2-dimensional (2D) view. UI element 304shows the user's viewpoint and/or position 306 on a map of the modelusing a dot, beacon, or a directional icon such as an arrow, etc. Inaddition, UI element 304 also serves as a means to navigate to adjacentor nonadjacent viewpoints or positions. For example, one way to navigateusing the 2D map interface is to click on a location on the map—thischanges the XY position within the model. When UI element 302 is updatedwith a new 3D view, the optimal rotation angle for the initial viewpointdirection or viewpoint vector may be preselected by the system. Therotation angle preselection may be based on factors such as presence offurther available translation points in the view, computationalefficiency, etc. For example, if a corner viewpoint location isselected, the initial viewpoint location may automatically be towardsthe center of the room rather than towards the corner. In FIG. 8, we seethe view of a tax parcel outline 802 and house outline 804. The user'sposition is shown with a directional icon 806 located on the parcel map,showing a viewpoint looking towards the house.

FIG. 9 shows a block diagram of the user interface 300 with an enhancedview of UI element 308. UI element 308 is a text overlay that broadcastsone or more labels associated with the user's location within thevirtual model—this is a 1-dimensional (1D) view—a linear list oflocations associated with the virtual model. UI element 308 tells theuser where in the virtual model the current viewpoint displayed in UIelement 302 is located, preferably based on spatial boundary names or alocation-related annotation in the virtual model. UI element 308 mayalso serve as a means to navigate to non-adjacent locations byincorporating a drop down menu 906. For example, one way to navigateusing the 1D interface would be to select a new location using the dropdown menu 906. Selecting a new location in drop down menu 906 wouldchange the current viewpoint position within the model based on apredetermined location and viewing vector associated with that label.FIG. 9 shows an embodiment of UI element 308 having a broadcast message902 indicating that the user is in the ‘Outdoors: Backyard’. Inaddition, this embodiment of UI element 308 has a dropdown element 904showing different locations that a user can jump to in the virtualmodel.

UI elements 302, 304, and 308, are interconnected based on the user'sviewpoint position within the virtual model, simultaneously broadcastinga 1D, 2D, and 3D view of the position based on position coordinates X,Y, Z, rotation, and pitch. Moreover, UI elements 302, 304, and 308 alsoserve as a navigation interface, enabling movement to differentpositions within the model by translation, rotation, or both.

In the next three examples, interaction between the three UI elements302, 304, and 308 are discussed in the context of remotely touring areal estate environment.

In one example, a user is looking at a viewpoint within the virtualmodel at a specific XY position with viewing vector rotation R aroundthe Z-axis and pitch U within a bathroom, next to the front door of aresidential house. In UI element 302, the users can see exactly what thebathroom looks like, based on an IBR rendering using a combination ofimage data and geometry. At the same time, the user can see in UIelement 304, the map view, that they are in the middle of the bathroomand that a number of different rooms and a hallway are adjacent to theirposition. Further, UI element 308 indicates that the user's position isGround Floor: Main Bathroom. To move to an adjacent viewpoint position,the user may select the desired location within UI element 302 and thencan further click and hold a mouse button while moving the mouse torotate their viewpoint within the bathroom in order to get differentperspectives of a faucet or shower in the bathroom. As soon asnavigation occurs in 3D based on interaction with UI element 302, UIelement 304 updates to show a new location on the 2D map of the bathroombased on the translation in 2D. Note that any rotation changes in 3Dwould not change the position in 2D shown on the map. At the same time,UI element 308 in this case would not change because the user'sviewpoint position was still within the same 2D spatial boundary, theGround Floor: Main Bathroom.

In another example, a user is viewing the virtual model from a specificperspective in the kitchen in UI element 302, seeing the room in 3D, asit would look from that position if they were physically present in thereal property. UI element 304 shows the user's 2D viewpoint location 2Dposition, which is a floor plan in this case. UI element 308 describesthe location as being indoors, on the ground floor, and in the kitchenby showing: “First Floor: Kitchen.” To move to a new viewpoint position,the user selects within UI element 104, perhaps choosing a new positionin the non-adjacent living room with new XY coordinates. Immediately,the 3D view in UI element 302 is updated to the new XY position with apredetermined viewing vector (i.e., predetermined rotation value andviewpoint pitch angle). Because UI element 302 also includes rotationinformation, the best or ideal viewing vector may be generatedautomatically for the rotation and pitch values based on a centroid ofthe density of samples, a connectivity graph, or other calculationsinvolving the geometry of the virtual model. At the same time, UIelement 308 also updates indicating that the user is now in First Floor:Living Room.

In another example of the linkage between the main user interfacecomponents, a user selects a viewpoint location outside of a residentialhouse, but within the boundary of the legal definition of the realproperty such as the parcel boundary. UI element 302 shows a 3D viewfrom a specific position and perspective from the backyard, lookingtowards the exterior of the house. UI element 304 shows a 2D map thatincludes the tax parcel boundary of the property, an outline of theexterior of the house, and an arrow indicating the user's viewpointposition in the backyard with a viewing vector pointing towards thehouse outline. UI element 308 broadcasts the users position as Outside:Backyard. To navigating to a new viewpoint location with UI element 308,the user selects a drop down menu in UI element 308, and selects thelink Second Floor: Master Bedroom Walk in Closet. Immediately, theuser's virtual location jumps to the new position inside the closet ofthe Master Bedroom, and the 3D view is shown in UI element 302. UIelement 304 updates by changing to a new map with a higher resolution(higher resolution due to fewer spherical panoramas captured outside ofthe house in this instance) showing a second floor floor plan, which isat a different elevation within the house; the user's viewpoint positionand 2D direction of the viewing vector are also shown on the map. UIelement 308 now broadcasts the previously selected “Second Floor: MasterBedroom Walk in Closet.” Note that in this example, the user has changedthe Z axis location of the viewpoint within the virtual model, eventhough only 4 DOF exist in the user interface of UI element 302.

Many parts of the User Interface System work together to create a uniqueuser experience for touring a real estate environment over theInternet—in particular the 3D view rendered in UI element 302 warrantsfurther discussion. In the 3D view of UI element 302, multiple IBRalgorithms can be combined to create a user experience that overcomesthe inherent tradeoff between photorealism and spatial navigation causedby using just one IBR algorithm exclusively. For example, densespherical panoramas may be combined with the use of view dependenttexture mapping (VDTM) during navigation as the user spatiallytranslates and rotates the viewpoint with the virtual model.

The capture process (defining a spatial boundary, data sampling, andannotation) for an enhanced user experience involves a dense sampling ofspherical panorama image data at multiple exposures (e.g., highdefinition rendering, HDR) and sampling of point cloud geometry from ahybrid camera and laser range scanning device. The processing pipeline(determining data sets, image calibration, image processing, 3Dreconstruction (3DR) and scene understanding) operates on the dataoutput from the capture process. Image calibration involves determiningpanorama pose and adding depth to each pixel of the panorama. Imageprocessing creates HDR spherical panoramas from input images. 3Dreconstruction involves removing noise from point clouds, reconstructinga real estate environment's geometry to varying degrees ofapproximation, generating geometric proxies that describe theenvironment with simpler meta primitives, feature matching betweenspherical panoramas, positioning of spherical panorama data in 3D space,feature matching between the panorama and 3D space, and compute viewdependent texture maps for the geometry and/or geometric proxies.

Rendering displays the processed data to an end-user via an IO device.During rendering, the user's position and navigation influence whichelements of geometry and image data are combined for a given Image-BasedRendering algorithm at any possible location in or around the virtualmodel. Ground truth image data is the captured image associated with aparticular capture location, and optionally may include any metadataassociated with the captured image, such as GPS coordinates, IR pointclouds, etc. Ground truth data may also include labels (outside:backyard, or groundfloor: bathroom) which are ground truth in the sensethat they are directly collected at the scene and are not syntheticapproximations. For example, when a user is near a position where groundtruth image data is captured, very little geometry is required to renderthe most photorealistic view of the model. At the exact position ofcapture, the use of an image primitive is the most photorealistic viewpossible. Composited image primitives such as a spherical panoramaenable 2 DOF rotational navigation. When translating directly betweentwo spherical panorama locations, other algorithms such as optical flowmay provide more photorealistic warping during the rendering ofpredetermined translational pathways defined in the connectivity graph.

When translating between other locations within the virtual model, theuse of VDTM over explicit geometric proxies combined with depth andfeature matching between nearby panoramas during rendering provides adecrease in photorealism but enables fluid movement to any spatiallocation. In contrast to traditional texture maps, VDTMs compute newtextures for different spatial positions, dynamically changing these asthe user's spatial position changes. This dramatically reduces artifactsassociated with appearance of objects.

In various embodiments, rendering may use RGB spherical panoramas, pointclouds, geometric proxies, view dependent texture mapping, and featurematching between the spherical panoramas and geometric proxies to createa fluid user experience involving multiple IBR algorithms thatdynamically change based on the user's location, direction, and speed ofmovement within the model. A user can view ground truth image data ornavigate to any other synthetic viewpoint in the same user interface,enjoying both photorealism and spatial navigation.

FIG. 10 shows a transaction pipeline associated with use of the UserInterface System. The transaction pipeline 1000 ties together the use ofIBR models of real property with specific transactions in differentbusiness scenarios. Business scenarios may involve selling, buying,renting, leasing, or booking real estate such as houses, apartments,offices, or hotels—all associated with real estate environments. IBRtechnology is ideally suited to commercial use for marketing in thesescenarios, because models are created with little or no manual work(excluding capture related tasks). Moreover, IBR approaches focus onrendering a photorealistic view of a real scene—exactly what is neededin real estate marketing to serve as a proxy for physically touring anenvironment.

A Site Map 1001 describes a website that is used to facilitatetransactions, market real estate, and render models of real estateenvironments which can be spatially navigated. Once on the website, theusers may select to Navigate 1002 an online model of a real estateenvironment. After interaction with the model, a user may initiate anumber of different business processes 1003 depending on the scenario,including but not limited to scenarios involving:

Residential real estate: ask a question, list a property, talk to anagent, schedule an in person tour, make an offer to buy, make an offerto buy contingent upon a traditional in-person home tour, or other tasksnecessary to coordinate and close residential transactions;

Commercial real estate: schedule an in person tour, make an offer tobuy, make an offer to lease, make an offer to lease or buy contingentupon an in-person tour, or other tasks necessary to coordinate and closecommercial transactions;

Real estate for rent: schedule an in person tour, make an offer to rent,make an offer to rent contingent upon an in-person tour, or other tasksnecessary to coordinate and close rental transactions; or

Hotels: book a hotel room or other tasks necessary to coordinate andclose hotel transactions.

Once a business process 1003 has been initiated the Transactioncomponent 804 supports various processes with workflow automationsoftware, web page screens, and mobile and tablet software that can beused over the Internet and hosted by the System. This workflowautomation software may streamline transactional details associated withone or more of selling, buying, leasing, renting, and bookingtransactions for real estate environments. In the case of residentialreal estate, these workflow tools may include:

A secure, real-time messaging platform between all parties to thetransaction: buyer, seller, listing agent, buyer's agent, lender, titleagent, escrow agent, appraiser, home inspector, and others etc.Collaborative features such as inviting other parties who are notimmediately connected to the transaction are enabled as well, such assending a real estate listing to a friend or asking an extended familymember to provide feedback on a house under consideration;

A timeline and contingency date tracking platform to ensure thattransaction details are executed and match the agreed upon contract;

A document tracking platform that unifies secure storage of alldocuments related to the transaction;

A term sheet platform that provides a simplified offer negotiationprocess where parties can agree to contract terms prior to drafting andsigning a complete and binding contract; or

An online negotiation platform that provides a collaborative environmentwhere a buyer and a seller can negotiate a term sheet or contract inreal-time without having to exchange copies of a contract.

In various embodiments of the transaction component 1004, electronicsignatures may be used to enable a paperless transaction.

Transaction component 1004 may include support for alternativetransaction types such as auctions, option contracts, lease to own,complicated contingencies, or making offers on off market propertiesthat are not currently listed.

In one example of a Transaction 1000, a potential home buyer issearching for homes using MLS data on a real estate brokerage website.Upon finding a home that meets their search criteria, the potentialbuyer reviews relevant information contained in the listing including alink to a virtual tour. Next, the potential buyer clicks on the virtualtour link, loading up a specific model associated with the listing.Navigating 802 through an online model, the buyer completes a remotehome tour. The business processes 803 are all focused on residentialreal estate sales, and the buyer can initiate a transaction 804 bycontacting the listing agent about the house through the system'smessaging interface. The potential buyer can also invite their spouse,relatives, buyer's agent, etc., to take the online tour as well bysending them a link to the model. The potential buyer may submit anoffer to buy the property via the system's messaging interface. Thehomeowner and listing agent would receive a notice of the offer via theterm sheet system and the homeowner could accept the offer via themessaging interface. Various embodiments of the transactional system 804may include messaging, calendaring, document tracking, electronicsignatures, and workflows across all parties necessary to completeclosing. For example, when coordinating a home inspection contingency,the transactional system 1004 would coordinate a mutually agreeable dateand time between the buyer, seller, buyer's agent and home inspector.Following inspection, the home inspector would upload the inspectionreport into the transactional system 804 which the buyer would reviewand then continue to move forward with the closing process. Othercontingencies and parties are similarly coordinated.

In another example of a transaction 1004, a visitor arrives on thewebsite 1001 and uses the UI 300 to navigate 1002 through an IBR modelof a real property. Impressed with the technology and value propositionto improve the availability of home tours over the Internet whilesimultaneously reducing the number of inconvenient visits from buyers,the visitor initiates the process 1003 to list a home for sale,accessing workflow software relevant to the transaction 1004. Oncecompleted, the data capture process is initiated to capture datanecessary for creating a virtual tour of the home, as well as otherrelevant downstream processes associated with the system necessary tomarket and sell the home.

In another example of a transaction process 1004, a potential homeseller could follow the listing process to activate an auction-stylelisting of their home where they could choose between multiple auctiontypes including an English, Dutch, sealed bid, or other approach. Thiscreates a predetermined auction-style business process 1003 which willdefine how potential buyers interact with the real estate listing. Aftercreation of a model, potential buyers can navigate the property via avirtual 3D tour and review any information included in the auction. Thepotential buyer may decide to submit a bid via the auction-styletransaction process 804, which initiates a series of closing relatedtasks for various parties.

In another example of a transaction process 1000, a potential homeseller could follow the traditional real estate listing process but alsoenable potential buyers to purchase a call option on the house bysetting up a call option business process 1003. A party interested inthe home could navigate 1005 through the property online and theninitiate an option agreement, paying an option fee to reserve apredetermined purchase price in the future. The owner and buyer wouldcollaborate on a term sheet prior to finalizing the document signaturesand any other transactional 1004 details would be coordinated to closethe deal.

FIG. 11 shows an embodiment of the user interface 300 with an enhancedtransaction capability. In the embodiment shown, items such as furnitureand appliances that are viewable via UI element 302 during a virtualtour may be associated with a transaction tag 1110 that indicates theitem may be commercially available for the viewer to purchase. Althoughthe transaction tag 1110 shown in FIG. 11 is perceptible by the viewer,transaction tags in some embodiments may be imperceptible by the user,such as annotations and metadata added during the data capture process.Other embodiments may not rely on transaction tags, instead relying onon-demand identification methods such as object, image, text, andtrademark recognition techniques to identify items shown in UI element302.

FIG. 12 shows a block diagram of an embodiment of an on-demand method ofproviding information about items in a virtual tour, including merchantinformation and advertisements via the user interface 100. For example,a potential home buyer may visit a real estate web site and take avirtual tour of a property listed for sale on the web site. During the3D tour of the home, user interface element 302 enables the viewer toselect objects seen in the virtual tour and receive information aboutthe objects. The information received can include the identity of localstores at which the object can be purchased, manufacturer of the object,etc.

In step 1202, the user selects an object that he sees in the virtualtour. For example, while touring the kitchen, the user may select thestove in step 1202. The on-demand system will retrieve the 3D image ofthe selected object from the virtual tour in step 1204 and search the 3Dimage for text and trademark images. After the object's 3D image and anyassociated text and trademarks have been identified, the systemconstructs a search query 1206 using object image data, extracted text,and extracted trademarks. In step 1208, the search query is sent to asearch engine which searches an object 3D image database to identify thetype of object (e.g., a cooking range) in step 1210. Location data mayalso be used to help identify the type of an object. For example, arectangular box-shaped object in the kitchen is more likely to be acooking range than a rectangular box-shaped object in the laundry room,which is more likely to be a washer or dryer. The brand (e.g.,manufacturer) of the object may be determined by searching the objectimage for identifying physical characteristics and searching a text andtrademark image database in step 1212. In step 1214, the results of theobject identification in step 1210 and the brand identification in step1212 are aggregated to identify the object and its manufacturer. Afterthe object has been identified, a database of local merchants issearched in step 1216 to identify nearby stores that sell the object. Inan alternative embodiment, the inventory of internet-based merchants maybe searched to identify websites from which the object may be purchased.In step 1218, the identifying information about the object is presentedto the user, along with information about merchants from whom the objectmay be purchased. In an alternative embodiment, an advertising engine(not shown) may provide an advertising placement opportunity for sale tomerchants who sell the product or competing products, thus allowing thesystem to provide highly relevant advertisements that correctly identifythe product the viewer/consumer wishes to purchase.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” Words using the singular or pluralnumber also include the plural or singular number respectively.Additionally, the words “herein,” “above,” “below” and words of similarimport, when used in this application, shall refer to this applicationas a whole and not to any particular portions of this application. Whenthe claims use the word “or” in reference to a list of two or moreitems, that word covers all of the following interpretations of theword: any of the items in the list, all of the items in the list and anycombination of the items in the list.

The above detailed descriptions of embodiments of the invention are notintended to be exhaustive or to limit the invention to the precise formdisclosed above. While specific embodiments of, and examples for, theinvention are described above for illustrative purposes, variousequivalent modifications are possible within the scope of the invention,as those skilled in the relevant art will recognize. For example, whilesteps are presented in a given order, alternative embodiments mayperform routines having steps in a different order. The teachings of theinvention provided herein can be applied to other systems, notnecessarily the embodiments described herein. These and other changescan be made to the invention in light of the detailed description.

These and other changes can be made to the invention in light of theabove detailed description. In general, the terms used in the followingclaims should not be construed to be limited to the specific embodimentsdisclosed in the specification, unless the above detailed descriptionexplicitly defines such terms. Accordingly, the actual scope of theinvention encompasses the disclosed embodiments and all equivalent waysof practicing or implementing the invention under the claims.

In view of the many possible embodiments to which the principles of thisinvention may be applied, it should be recognized that the detailedembodiments are illustrative only and should not be taken as limitingthe scope of the invention. Thus, we claim as our invention all suchembodiments as may come within the scope and spirit of the followingclaims and equivalents thereto.

1. A method embodied in a computer system, the method comprising:maintaining a web site system that facilitates selection of a realproperty from a database of real properties; via the user interface,receiving from a visitor a first selection, the first selectionindicating a real property which the visitor would like to virtuallytour; via the user interface, receiving from the visitor a request toinitiate a virtual real estate tour; via the user interface, displayinga 3D viewpoint within a virtual model of the real property; via the userinterface, displaying a 2D map overlay of the real property on a firstportion of the user interface; and via the user interface, displaying a1D text overlay of the real property on a second portion of the userinterface.